{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML para la Detección de Malware en Cabeceras de Ficheros PE\n",
    "\n",
    "Referencias:\n",
    "- Libro [\"Mastering Machine Learning for Penetration Testing\"](https://www.packtpub.com/eu/networking-and-servers/mastering-machine-learning-penetration-testing) de Chiheb Chebbi.\n",
    "- \"Machine Learning for Malware Detection\" por Cristi Vlad."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ficheros PE\n",
    "\n",
    "<blockquote>El formato Portable Executable (PE) es un formato de archivo para archivos ejecutables, de código objeto, bibliotecas de enlace dinámico (DLL), archivos de fuentes FON, y otros usados en versiones de 32 bit y 64 bit del sistema operativo Microsoft Windows. El término \"portable\" refiere a la versatilidad del formato en numerosos ambientes de arquitecturas de software de sistema operativo. El formato PE es una estructura de datos que encapsula la información necesaria para el cargador de Windows para administrar el código ejecutable que le envuelve. </blockquote>\n",
    "\n",
    "https://es.wikipedia.org/wiki/Portable_Executable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requisitos:\n",
    "\n",
    "- Python 3\n",
    "- Paquete Scikit-Learn\n",
    "- Paquete pandas\n",
    "- Paquete \"pefile\" (Portable Executable reader module):\n",
    "    - `pip install pefile`\n",
    "    - `conda install -c conda-forge pefile`\n",
    "- Graphviz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datos\n",
    "\n",
    "Malware dataset: https://github.com/chihebchebbi/Mastering-Machine-Learning-for-Penetration-Testing/blob/master/Chapter03/MalwareData.csv.gz\n",
    "\n",
    "- 41,323 binaries (exe, dll) - legítimos\n",
    "- 96,724 ficheros malware de virusshare.com\n",
    "\n",
    "Fuente: \"Mastering Machine Learning for Penetration Testing\" by Chiheb Chebbi.\n",
    "\n",
    "<div style=\"text-align: center\"><img src=\"https://www.packtpub.com/media/catalog/product/cache/e4d64343b1bc593f1c5348fe05efa4a6/b/1/b10116.png\" style=\"width: 100px;\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Análisis de los datos mediante pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv(\"MalwareData.csv\", sep=\"|\")\n",
    "\n",
    "print(\"El fichero contiene %d ejemplos.\"%data.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_columns\",None) # visualizamos todas las columnas\n",
    "print(data.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.take([12])) # visualizamos todas las caracetrísticas de un dato"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Separación de los datos en train y test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = data.drop([\"Name\",\"md5\",\"legitimate\"],axis=1).values\n",
    "y = data['legitimate'].values\n",
    "\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# separamos los datos en 2 conjuntos\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)\n",
    "\n",
    "print(X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reducción del número de características"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inicializamos el clasificador\n",
    "clf = RandomForestClassifier(n_estimators=50,max_depth=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# entrenamos el clasificador\n",
    "clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# seleccionamos las características más relevantes basado en sus pesos\n",
    "selected_features = SelectFromModel(clf,prefit=True)\n",
    "\n",
    "# reducimos las tablas de los datos\n",
    "X_train_reduced = selected_features.transform(X_train)\n",
    "X_test_reduced = selected_features.transform(X_test)\n",
    "\n",
    "print(X_train_reduced.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Análisis de las características seleccionadas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "n_features = X_train_reduced.shape[1]\n",
    "importances = clf.feature_importances_\n",
    "indices = np.argsort(importances)[::-1]\n",
    "\n",
    "feature_names = data.columns[2:] # anteriormente hemos quitado las primeras 2 características\n",
    "feature_names_reduced = feature_names[indices[0:n_features]]\n",
    "\n",
    "print(feature_names_reduced)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Entrenamos un clasificador Random Forest en los datos reducidos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inicializamos\n",
    "clf = RandomForestClassifier(n_estimators=50,max_depth=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# entrenamos\n",
    "clf.fit(X_train_reduced,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Puntuación del algoritmo: %.2f%%\"%(clf.score(X_test_reduced,y_test)*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# comprobación de falsos negativos y falsos positivos\n",
    "\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "y_pred = clf.predict(X_test_reduced)\n",
    "conf_mat = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "print(conf_mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# matrix de confusión usando porcentajes\n",
    "confusion_matrix(y_test, y_pred)/X_test.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Falsos positivos: %.2f%%\"%(conf_mat[0][1]/sum(conf_mat[0])*100))\n",
    "print(\"Falsos negativos: %.2f%%\"%(conf_mat[1][0]/sum(conf_mat[1])*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualización de un árbol del Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import export_graphviz\n",
    "\n",
    "estimator = clf.estimators_[0]\n",
    "\n",
    "# exportamos como fichero dot\n",
    "export_graphviz(estimator, out_file='tree.dot', \n",
    "                feature_names = feature_names_reduced,\n",
    "                class_names = ['malware','legitimate'],\n",
    "                rounded = True, proportion = False, \n",
    "                precision = 2, filled = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convertimos a png (require Graphviz)\n",
    "from subprocess import call\n",
    "call(['dot', '-Tpng', 'tree.dot', '-o', 'tree.png', '-Gdpi=600'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualización\n",
    "from IPython.display import Image\n",
    "Image(filename = 'tree.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
